The Python Data Science Stack¶
Presidents of the USA¶
The Python code below will load a dataset containing the names of the first 44 presidents of the USA and their heights, available in the file president_heights.csv, which is a simple comma-separated list of labels and values.
# Imports
import numpy as np
import pandas as pd
from pandas import DataFrame, Series
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import pearsonr
file = 'president_heights.csv'
presidents = pd.read_csv(file)
presidents
The below code will display the histogram of president's heights and compute summary statistics:
- Mean height
- Standard deviation
- Minimum height, and
- Maximum height.
height = np.array(presidents['height(cm)'])
plt.hist(height)
plt.title('Height Distribution')
plt.xlabel('Height')
plt.ylabel('Number of Presidents');
plt.show()
print("Height statistics")
print("Average height (global):", np.mean(height))
print("Standard Deviation of Height (global):", np.std(height))
print("Minimum height (global):", np.min(height))
print("Maximum height (global):", np.max(height))
Nxt we will write Python code to answer the following questions:
- Who was(were) the tallest president(s)?
- Who was(were) the shortest president(s)?
- How many presidents were 6' tall or taller?
print("Tallest preisdent(s): ")
for i in range(0, len(height)):
if height[i]==np.max(height):
print(str(presidents["order"][i]) + " " +presidents["name"][i])
print("Shortest preisdent(s): ")
for i in range(0, len(height)):
if height[i]==np.min(height):
print(str(presidents["order"][i]) + " " +presidents["name"][i])
height_in_ft=height/30.48
print("Number of Presidents over 6 feet tall: ", len([x for x in height_in_ft if x>=6]))
height
This is an extremely small, simple and manageable dataset.
Let's use it to prove a silly hypothesis, for example:
"H1: Even-numbered presidents are, in average, taller than odd-numbered ones."
odd_height=height[0::2]
even_height=height[1::2]
if np.mean(even_height)>np.mean(odd_height):
H1=True
else:
H1=False
H1
Hypothesis H1 was refuted.
Next we will text Hypothesis H2:
H2: The first 22 presidents are, on average, shorter than the last 22.
first_22=height[0:21:1]
last_22=height[22:43:1]
if np.mean(first_22)<np.mean(last_22):
H2=True
else:
H2=False
H2
Hypothesis H2 was confirmed by the data.
HR payroll¶
The Python code below will load a dataset containing the salaries and demographic data of more than 1000 employees of a hypothetical company, available in the file salaries.csv, which is a simple comma-separated list of labels and values.
salaries = pd.read_csv('salaries.csv')
print(salaries.shape)
print(salaries.count())
earn= employee salary
height= employee height
sex= employee sex
ed= years of employee education
age= employee age
race= employee race
Let's explore the dataset by plotting some graphs and displaying summary statistics.
The code below should display:
- Min, max, average, and median salary (global)
- A histogram of salaries
- A scatterplot correlating salaries and years of education
- The (Pearson) correlation coefficient between the two variables.
This should help us get started.
salary = np.array(salaries['earn'])
print("Salary statistics")
print("Minimum salary (global):", np.min(salary))
print("Maximum salary (global):", np.max(salary))
print("Average salary (global):", np.mean(salary))
print("Median salary (global):", np.median(salary))
plt.hist(salary)
plt.title('Salary Distribution')
plt.xlabel('Salary')
plt.ylabel('Number of Employees');
plt.show()
years = np.array(salaries['ed'])
plt.title('Salary vs. Education Level')
plt.xlabel('Salary')
plt.ylabel('Years of education');
plt.scatter(salary, years, alpha=0.5)
plt.show()
# Compute Pearson coefficient
from scipy.stats import pearsonr
corr, _ = pearsonr(salary,years)
print('Correlation coefficient: ',corr)
The Pearson correlation coefficient (a value between -1 and 1) can be used to summarize the strength of the linear relationship between two data samples.
A simplified way to interpret the result is:
- A value of 0 means no correlation
- Values below -0.5 or above 0.5 indicates a notable (negative/positive) correlation
The code below should:
- Display the total headcount and the number (and %) of male and female employees.
- Compute and display the min, max, average, and median salary per gender.
- Display a boxplot that could provide insight into the gender inequality (if any is present) associated with the salaries in the company.
genders=np.array(salaries['sex'])
males=[]
females=[]
for g in genders:
if g=='male':
males.append(g)
else:
females.append(g)
males=np.array(males)
females=np.array(females)
total_headcount=len(males)+len(females)
print("number of males:", len(males), (len(males)/total_headcount)*100)
print("number of females:", len(females), (len(females)/total_headcount)*100)
male_salaries=[]
female_salaries=[]
for i in range(0, len(salaries)):
if salaries['sex'][i]=='male':
male_salaries.append(salaries['earn'][i])
else:
female_salaries.append(salaries['earn'][i])
male_salaries=np.array(male_salaries)
female_salaries=np.array(female_salaries)
print("Salary statistics per gender:")
print("Minimum salary (male):", np.min(male_salaries))
print("Maximum salary (male):", np.max(male_salaries))
print("Average salary (male):", np.mean(male_salaries))
print("Median salary (male):", np.median(male_salaries))
print("Minimum salary (female):", np.min(female_salaries))
print("Maximum salary (female):", np.max(female_salaries))
print("Average salary (female):", np.mean(female_salaries))
print("Median salary (female):", np.median(female_salaries))
salaries.boxplot(column='earn', by='sex')
plt.title("Salaries by Sex")
plt.suptitle('')
plt.show()
It is clear from the dual boxplot above that the male group generally earns significantly more than the female group.
As you can possibly tell by now, this dataset may help us test hypotheses and answer questions related to possible sources of inequality associated with the salary distribution: gender, age, race, height.
Let's assume, for the sake of argument, that the number of years of education should correlate well with a person's salary (this is clearly a weak argument and the plot and Pearson correlation coefficient computation above suggests that this is not the case) and that other suspiciously high (positive or negative) correlations could be interpreted as a sign of inequality.
At this point, we formulate 3 different hypotheses that might suggest that the salary distribution is biased by factors such as ageism.
Call these hypotheses H3, H4, and H5.
H3: Older employees (65 and older) make less on average than younger ones.
H4: Non-white employees make less on average than white employees.
H5: Shorter employees make less on average than taller ones.
Next we will write Python code to test hypotheses H3, H4, and H5 (and some text to explain whether they were confirmed or not).
young_salaries=[]
older_salaries=[]
for i in range(0, len(salaries)):
if salaries['age'][i]<65:
young_salaries.append(salaries['earn'][i])
else:
older_salaries.append(salaries['earn'][i])
young_salaries=np.array(young_salaries)
older_salaries=np.array(older_salaries)
if np.mean(older_salaries)<np.mean(young_salaries):
H3=True
else:
H3=False
print(H3)
plt.scatter(salaries['age'], salaries['earn'])
plt.title("Age vs. Salary")
plt.xlabel("Age(years)")
plt.ylabel("Salary")
plt.show()
Hypothesis H3 was confirmed. Senior employees (65 and older) make less on average than younger employees. However, examining the scatter plot above reveals that much of the lower paid employees are actually more on the younger side of the age range of this dataset, with the highest paying jobs concentrated around the middle. This makes intuitive sense as middle aged employees are often in the prime of their career.
white_salaries=[]
nwhite_salaries=[]
for i in range(0, len(salaries)):
if salaries['race'][i]=='white':
white_salaries.append(salaries['earn'][i])
else:
nwhite_salaries.append(salaries['earn'][i])
white_salaries=np.array(white_salaries)
nwhite_salaries=np.array(nwhite_salaries)
if np.mean(nwhite_salaries)<np.mean(white_salaries):
H4=True
else:
H4=False
print(H4)
salaries.boxplot(column='earn', by='race')
plt.title("Salaries by Race")
plt.suptitle('')
plt.show()
Hypothesis H4 was confirmed. Nonwhite employees make less on average than white employees. In addition, it is clear from the boxplots above that there is an inherent salary bias towards white employees in this company. Although there are much fewer non-white employees in this company, their salaries come nowhere near those of white employees.
short_salaries=[]
tall_salaries=[]
for i in range(0, len(salaries)):
if salaries['height'][i]<66:
short_salaries.append(salaries['earn'][i])
else:
tall_salaries.append(salaries['earn'][i])
short_salaries=np.array(short_salaries)
tall_salaries=np.array(tall_salaries)
if np.mean(short_salaries)<np.mean(tall_salaries):
H5=True
else:
H5=False
print(H5)
plt.scatter(salaries['height'], salaries['earn'])
plt.title("Height vs. Salary")
plt.xlabel("Height (inches)")
plt.ylabel("Salary")
plt.show()
Hypothesis H5 was confirmed. On average, shorter employees (shorter than 66 inches) make less than taller ones. In addition, an examination of the scatter plot above reveals that the highest paid employees are near the median value for height.
Fuel consumption¶
The Python code below will load a dataset containing fuel consumption data for ~400 vehicles produced in the 1970s and the 1980s along with some characteristic information associated with each model.
Here, displacement refers to a vehicle's engine size and the fuel efficiency is measured in miles per gallon (mpg).
See: https://archive.ics.uci.edu/ml/datasets/Auto+MPG for additional information.
sns.set(style='ticks', palette='Set2')
%matplotlib inline
data = pd.read_csv("http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data-original",
delim_whitespace = True, header=None,
names = ['mpg', 'cylinders', 'displacement', 'horsepower', 'weight', 'acceleration',
'model', 'origin', 'car_name'])
print(data.shape)
data.dropna(inplace=True)
data.head()
data.reset_index(drop=True, inplace=True)
data
The code below should:
- Count the number of 3- and 5-cylinder vehicles in the dataset, display the count, and discard those entries (rows).
- Compute and display the min, max, and average fuel consumption (in mpg) for 4-, 6-, and 8-cylinder vehicles.
- Display the name of the most and least fuel efficient vehicles in the dataset.
cylinder_values=np.array(data['cylinders'])
cylinder_count=0
for i in range(0, len(cylinder_values)):
if cylinder_values[i]==3.0 or cylinder_values[i]==5.0:
cylinder_count=cylinder_count+1
print("Number of 3 and 5-cylinder vehicles: ", cylinder_count)
data=data.drop(data[(data.cylinders ==3.0) | (data.cylinders ==5.0)].index)
data
fuel_consumption=np.array(data['mpg'])
fuel_max=np.max(fuel_consumption)
fuel_min=np.min(fuel_consumption)
fuel_avg=np.mean(fuel_consumption)
print("Maximum mpg: ", fuel_max)
print("Minimum mpg: ", fuel_min)
print("Average mpg: ", fuel_avg)
print("Most fuel efficient vehicle(s): ")
for i in range(0, len(fuel_consumption)):
if fuel_consumption[i]==np.max(fuel_consumption):
print(data['car_name'][i]+ "\n")
print("Least fuel efficient vehicle(s): ")
for i in range(0, len(fuel_consumption)):
if fuel_consumption[i]==np.min(fuel_consumption):
print(data['car_name'][i]+ "\n")
This dataset may help us test hypotheses and answer questions related to fuel consumption.
To get started: Which features of a vehicle correlate best with its mpg -- displacement, weight, or horsepower?
The Python code below should plot the relationship between:
- Fuel consumption and displacement (engine size)
- Fuel consumption and weight
- Fuel consumption and horsepower (HP)
plt.scatter(data['mpg'], data['displacement'])
plt.title("Fuel Consumption vs. Displacement")
plt.xlabel("Fuel consumption(mpg)")
plt.ylabel("Displacement")
plt.show()
plt.scatter(data['mpg'], data['weight'])
plt.title("Fuel Consumption vs. Weight")
plt.xlabel("Fuel consumption(mpg)")
plt.ylabel("Weight")
plt.show()
plt.scatter(data['mpg'], data['horsepower'])
plt.title("Fuel Consumption vs. Horsepower")
plt.xlabel("Fuel consumption(mpg)")
plt.ylabel("Horsepower")
plt.show()
There is a negative correlation between mpg and displacement. It appears to be linear.
There is a negative correlation between mpg and weight. It appears to be non-linear.
There is a negative correlation between mpg and horsepower. It appears to be non-linear.
Next we will write Python code to produce box plots that should provide good answers the questions below:
- Did vehicles get more efficient over the years (represented in this dataset, i.e., 1970 through 1982)?
- Are Japanese cars more fuel efficient than American or European ones?
data['Country_code'] = data.origin.replace([1,2,3],['USA','Europe','Japan'])
data.boxplot(column='mpg', by='model')
plt.title('')
plt.suptitle("Fuel efficiency by year")
plt.ylabel("Fuel consumption(mpg)")
plt.show()
data.boxplot(column='mpg', by='Country_code')
plt.title('')
plt.suptitle("Fuel efficency by countries")
plt.ylabel("Fuel consumption(mpg)")
plt.show()
The two boxplots above show that there is a genereal trend of increasing fuel efficiency as time moves forward.
In addition, it is evident that Japanese cars are generally more fuel efficient than American cars.